feat(wasm): add WebAssembly build and @columnar-tech/dbc-wasm npm package - #395
Conversation
amoeba
left a comment
There was a problem hiding this comment.
Thanks for this. I did a quick scan and found one issue related to credential paths.
As an aside, which version of dbc will we recommend people use for Node.js?
|
|
||
| package config | ||
|
|
||
| // splitConfigList does not split under GOOS=js: the wasm API passes a single |
There was a problem hiding this comment.
Is this really needed? Since SplitList only ever runs on a single platform, mixing isn't an issue? Not sure if a change is needed here but just commenting.
There was a problem hiding this comment.
It is, but not because of platform mixing — it's what GOOS=js does to filepath. Under js, filepath.ListSeparator is : (js is treated as a Unix target regardless of the actual host OS), so on a Windows host a drive-lettered location like C:/drivers would split into ["C", "/drivers"]. The wasm API always passes a single explicit directory, so the correct behavior under js is "don't split," which is what this override does; native builds keep the normal filepath.SplitList in splitlist_other.go.
Compile dbc to GOOS=js GOARCH=wasm and expose a Node-importable library covering search, install-to-disk, list, uninstall, and signature verification, plus private-registry authentication. - telemetry: build-tagged telemetryMachineID() returns "" under js (machine-id has no wasm port); removed its direct import from client.go and drivers.go - client: export WithCredentialResolver for host-injected credentials - config: SetPlatformTupleOverride (js-only) sets the host platform tuple, since GOARCH=wasm otherwise yields unknown_wasm64 - wasm/: fetch-backed http.RoundTripper, a Promise+goroutine bridge, and the Node entrypoint (search/install/list/uninstall/verify). The entrypoint routes http.DefaultClient through fetch so dbc's auth refresh, license, and OIDC calls work under Node - wasm/spike: Node harnesses, report, and run script validating the gate end-to-end (config paths, search over fetch, install/list/ uninstall on disk, gopenpgp signature verify, 401 -> refresh -> retry, and no event-loop deadlock under concurrency) Native build and dbc/config/auth tests are unaffected.
… test Refactor the wasm entrypoint so a future browser build is a cheap addition and shares logic with the Node build: - api_js.go (js): shared handlers (search/resolve/verify), client setup, JS config hooks, and the http.DefaultClient -> fetch override - ops_node_js.go (js && dbcnode): install/list/uninstall (filesystem) - main_node.go / main_browser.go: thin entrypoints. The browser build omits ops_node_js.go entirely, so install/extract code is excluded (browser binary is ~338 KB smaller than the node build) - add resolve(name, platform) returning versions + latest package URL - add wasm/dbc.d.ts TypeScript declarations - add native unit tests for WithCredentialResolver Verified: native build + dbc/config tests pass; vet clean for native and both js builds; node harness regression green.
… race
- search/resolve (wasm/api_js.go): no longer reject the Promise on a partial
Client.Search error (one registry fails while another succeeds). Search now
returns {drivers, warning} and rejects only when zero drivers resolve,
mirroring the CLI --json warning contract; resolve proceeds with partial
results so a reachable registry still answers
- list (wasm/ops_node_js.go): use the new env-free config.FindDriverConfigsIn
so concurrent dbcList calls no longer race on the process-global
ADBC_DRIVER_PATH
- config: add FindDriverConfigsIn(location) plus a native unit test
- update dbc.d.ts (SearchResult) and the spike harnesses for the new
search return shape
… 6510) jsResolve guarded the early-return on a partial Client.Search error, but when the exact driver was absent from the partial results it fell through to a plain "not found" that hid the registry failure. Search does substring matching, so a reachable registry's non-exact hits could mask an outage on the registry hosting the exact driver. Now wrap searchErr in the not-found error; return plain "not found" only when search completed cleanly.
Node-importable packaging for the wasm build, separate from the native-binary @columnar-tech/dbc wrapper. - loader (index.cjs + index.mjs): wires Node fs/process/webcrypto and go.env=process.env before wasm_exec.js (the two Phase 0 R1 requirements), instantiates dbc.wasm, and exposes an async API (search/resolve/install/ uninstall/listInstalled/verifySignature) that auto-detects the host platform tuple and JSON-parses results - index.d.ts: TypeScript declarations for loadDbc plus result types - scripts/build.js: compiles ./wasm (-tags dbcnode), copies wasm_exec.js from GOROOT, generates package.json at a given version, copies LICENSE; build artifacts are gitignored, mirroring the packages/npm conventions - test/smoke.cjs: end-to-end smoke test through the package API against the repo test fixtures - README documents usage, the Node >=18 requirement, and credential bootstrap Verified: build + smoke pass (search/resolve/install/list/verify/uninstall); ESM and CJS entrypoints load; npm pack yields an 8-file ~4 MB tarball.
…roborev 6519)
loadDbc previously applied baseURL/credential by mutating singleton wasm
globals, so two clients in one Node process clobbered each other: after a
second loadDbc, the first client's search/install would use the later
baseURL/credential — a cross-tenant registry/credential mixup.
- api_js.go: replace the baseURL/credResolver globals and the dbcSetBaseURL/
dbcSetOAuthCredential setters with clientFromConfig(cfgJSON); dbcSearch and
dbcResolve take a leading config JSON and build the dbc.Client per call
- ops_node_js.go: dbcInstall takes the config JSON; dbcUninstall uses a bare
client (it performs no network I/O)
- index.cjs: the loader captures baseURL+credential per loadDbc instance and
passes them to each network call; platform is set once at init as a
process-global host constant (overridable via loadDbc({ platform }))
- spike harnesses updated for the cfg-first signatures
Verified: two clients with different baseURLs stay isolated (a.search() still
hits registry A after b=loadDbc with baseURL B); package smoke and the auth
401 -> refresh -> retry harness pass; native build/tests + vet clean.
The fetch RoundTripper buffered each response via arrayBuffer(), holding the full driver tarball in wasm linear memory before copying it to disk. Replace that with a ReadableStream-backed io.ReadCloser (jsStreamBody) so install copies the tarball to disk chunk-by-chunk; ContentLength is taken from the header and null bodies map to http.NoBody. Read awaits the stream reader, so it runs on the existing per-call goroutine. Verified: package smoke + spike harnesses (search/install/verify/uninstall) and the auth 401 -> refresh -> retry harness pass; node and browser builds clean.
…orev 6521) The roborev-6519 fix rebuilt a fresh auth.Credential from the immutable config JSON on every call, so a 401 -> refresh updated only that throwaway credential; the next call started from the original (stale) token again — wasteful, and fatal with single-use refresh tokens (operations 2+ would fail). - api_js.go: keep a per-instance dbc.Client in a handle registry. dbcNewClient(cfg) builds the client once and returns an int handle; search/resolve/install take the handle and reuse the stored client, so the credential pointer (and any refreshed access token) persists across calls. dbcCloseClient releases it. - ops_node_js.go: dbcInstall takes the handle. - index.cjs: loadDbc creates a handle, exposes close(), and registers a FinalizationRegistry to release handles on GC. - index.d.ts: add Dbc.close(); spike harnesses updated (the auth harness now asserts a single token refresh across two searches). Verified: tokenRefreshes==1 across two searches with a stale injected token; two clients stay isolated (6519); package smoke + native tests + vet clean.
…ase 4) Under GOOS=js, path/filepath uses Unix semantics on every host, so filepath.SplitList splits on ":" and corrupts a Windows drive-lettered location like C:/drivers. Per an Oracle architecture review, introduce a build-tagged splitConfigList() seam (non-js: filepath.SplitList byte-for-byte; js: the single explicit location, no split) and use it at the explicit-location sites the wasm API traverses: - config.EnsureLocation (install) and config.GetDriver env branch (uninstall) - config.FindDriverConfigsIn (list) loadConfig/getEnvConfigDir (env-var multipath, out of wasm scope) are left alone. filepath.Join/Dir/Base/Clean/OpenRoot are already safe under js (Node fs accepts forward-slash drive paths); filepath.IsAbs is not on the install path. The npm loader also normalizes a Windows location at the JS boundary (backslashes to forward slashes; drive-relative to absolute). Native builds are byte-identical (splitConfigList aliases filepath.SplitList off js); native + Linux-wasm tests/smoke/spike are unchanged. NOT yet verified on a Windows runtime: os.MkdirAll/os.OpenRoot over a drive root through Node fs (see README caveats) -- validate on a Windows runner before claiming support.
…roborev 6551) The in-process and worker backends each constructed their own client object literal and had drifted: sync vs async close(), always-on vs feature-detected install methods, and prefixed vs raw error messages. Factor both behind a single buildClient() factory driven by a per-backend call(fn, args) dispatcher so the public shape is defined once and cannot diverge. - close() now returns Promise<void> and is idempotent in both backends; the in-process path guards against a double-free and unregisters the FinalizationRegistry on explicit close. - install/uninstall/listInstalled are feature-detected symmetrically: the worker reports hasInstall in its ready handshake, matching the in-process typeof check. - All surfaced errors are namespaced with "dbc-wasm:" via a shared prefixError() helper (worker RPC rejections and in-process Go errors included). - index.d.ts/README document the remaining design realities rather than changing behavior: mandatory close() under worker:true (worker thread is not GC-managed), process-global platform vs per-instance baseURL/credential and resolve()'s per-call precedence, hostPlatformTuple()'s throw, normalizeLocation as a low-level internal export, the search-only warning rationale, a DbcClient alias, verifySignature(library, signature) param names, and a comment on the install-vs-uninstall handle asymmetry. Verified by the normalize, worker-smoke, and in-process-smoke tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The runtime instantiation sequence (inject fs/process/crypto globals, load
wasm_exec.js, instantiate dbc.wasm, run, and verify the dbc* API registered)
was duplicated byte-for-byte between index.cjs (ensureRuntime) and worker.cjs.
Any change to the boot protocol had to land in both, and a divergence would be
a silent init bug caught only by whichever path was exercised.
Extract it into a single bootRuntime() in boot.cjs. Each caller keeps its own
error-reporting policy: the in-process path namespaces the rejection with the
dbc-wasm: prefix, the worker posts a {type:"fatal"} message. Add boot.cjs to
the published files list in the build.js package.json generator so it ships in
the tarball.
Smoke tests pass for the in-process, worker, and worker-init-failure paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lper The worker backend repeated the `closed = true` toggle and pending-RPC rejection inline across five sites (the message/fatal, error, exit handlers, the init catch block, and close()). Extract a single idempotent markClosed(err) helper that flips `closed`, drains `pending`, and returns whether it won the teardown race. The event handlers and close() now call one function instead of re-implementing the guard, removing the "did I guard `closed` here?" class of bug (R3 / smells #3, #4). Behavior is unchanged: the exit handler still skips readyReject when an explicit close()/init-failure already tore down (now via markClosed's return value), and the init catch still terminates the worker before propagating. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The init gate (`readyReject`) was rejected separately from the in-flight RPCs (`markClosed`) in every worker failure handler, so a reader had to hold two rejection channels in mind to confirm a failing worker can neither hang nor double-settle. Route `readyReject` through markClosed so every failure path settles both channels at once and each handler becomes a one-liner. Behavior is unchanged (already-settled promises ignore later rejects; the `closed` guard short-circuits repeats), verified by the existing worker smoke test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…(roborev 6562) loadDbc()'s in-process path created the client via globalThis.dbcNewClient(cfg) outside the call() dispatcher, so load-time failures (e.g. an invalid credential registryURL) rejected without the dbc-wasm: namespace the worker backend and the rest of this layer apply. Wrap it in try/catch and rethrow prefixError(e) for consistent error attribution.
…orev 6563) Adds an in-process regression assertion to smoke.cjs mirroring the worker path: loadDbc() with an invalid credential registryURL must reject with a dbc-wasm:-prefixed message, locking in the error-attribution fix from 0046b19.
- Delete stale/unreferenced wasm/dbc.d.ts; packages/npm-wasm/index.d.ts is the single canonical type declaration. - Document main_browser.go as an experimental, not-yet-delivered future-phase seam (no loader/npm entry/test). - REPORT.md: superseded change-set note for the main_node.go -> api_js/ops_node_js/bridge_js/roundtripper_js split; Go 1.26.4 (spike) vs CI-pinned 1.25; Phases 1-2 absorbed into Phase 3/4. - README: reconcile the Windows caveat with the windows-latest CI smoke gate (define the signal instead of 'not verified'). - index.d.ts: document the in-process process-global platform limitation for install/listInstalled, and the SignedByColumnar-only verifySignature trust model.
…ndows boot.cjs set go.env = process.env, which on Windows CI runners (large environments) overflowed wasm_exec.js's combined argv+env size cap, throwing 'total length of command line and environment variables exceeds limit' from go.run() before the runtime started. Forward only the vars the GOOS=js runtime needs ($HOME, $XDG_*, temp dir), mapping Windows %USERPROFILE% to $HOME. Linux/macOS smoke+worker tests unaffected.
…ows (roborev 6570) Go built with GOOS=js resolves os.TempDir() from $TMPDIR only (falling back to /tmp), so the prior curation that forwarded %TEMP%/%TMP% under their own names left install creating a temp dir under a nonexistent /tmp on Windows. Map TMPDIR from TMPDIR||TEMP||TMP (and HOME from HOME||USERPROFILE), normalizing backslashes to forward slashes on Windows. Extract the curation into a pure curateGoEnv(env, platform) and cover it with deterministic unit tests in normalize.test.cjs.
Diagnostic for the windows-latest listInstalled failure: include the actual listInstalled() result and a recursive listing of the install directory in the assertion message so the Windows CI log shows where install output and driver discovery diverge.
…dows hosts loadDir enumerated manifests with fs.Glob(os.DirFS(dir), "*.toml"), but under GOOS=js an os.DirFS rooted at a Windows drive path (e.g. C:/drivers) matches nothing, so listInstalled returned no drivers on Windows hosts even though the manifests were present on disk (caught by the WASM windows-latest smoke). Enumerate with os.ReadDir(dir) + a .toml suffix filter, which goes straight through the host fs and is behavior-equivalent on native. Native config tests + Linux wasm smoke/worker pass.
The WASM Windows-host path is experimental and not yet passing: driver discovery (listInstalled) returns empty under GOOS=js for drive-letter locations even after install succeeds (tracked in #396). Exclude windows-latest from the WASM workflow matrix until fixed, revert the loadDir os.ReadDir change (it did not resolve the Windows discovery issue), and reconcile the README to state Windows WASM is experimental and not CI-gated. The env-overflow and TMPDIR loader fixes remain.
Document that omitting baseURL uses Columnar's public driver CDN (https://dbc-cdn.columnar.tech) and that setting it points at a private/self-hosted registry.
…age + CI) wasm/spike/ (REPORT.md + Node harnesses + run.sh) was a Phase 0 proof of concept. The feature now ships as @columnar-tech/dbc-wasm with a CI-gated test suite (smoke/worker/normalize), so the spike is superseded. Not referenced by CI, docs, or code; history retained in git.
…tream bug) Bun's WebAssembly <-> node:fs callback bridge mishandles values from Go's GOOS=js runtime, so file-writing ops (install/uninstall) fail with ERR_OUT_OF_RANGE. Node >= 18 and Deno are validated; Bun is unsupported pending an upstream fix.
Add a SHA-pinned denoland/setup-deno step and run normalize/smoke/worker under 'deno run -A' after the Node steps (ubuntu + macos). Deno is a validated runtime; this gates it in CI.
Co-authored-by: Bryce Mecum <petridish@gmail.com>
The applied review suggestion left the description string unterminated (trailing comma, no closing quote), breaking 'node build.js' with a SyntaxError and failing the Wasm CI build on ubuntu/macos. Close the string and keep the reviewer's intended wording.
- rename workflow to 'Wasm'
- match Go sources recursively ('**/*.go') so changes in any package
trigger the build
- bump pinned actions to current refs: actions/checkout v6.0.2 (matches
dev.yml) and actions/setup-go v6.5.0
go.mod requires go >= 1.26; install Go 1.26 so the wasm build works under setup-go v6 (which pins GOTOOLCHAIN=local and so won't auto-download a newer toolchain).
0397127 to
6c2ca74
Compare
It's a question of use case. Similar to the question of whether they should use the You only really need to import the library (in this case the wasm) if you're intending to integrate dbc into your application and dynamically manage drivers at runtime but don't want to (a) bundle the dbc CLI alongside your application as a sidecar or (b) shell out to exec dbc as a separate process that you have to parse the output from. The idea is that you can just use the dbc CLI directly (in node/deno you can do Does that make sense? |
|
It makes sense but I'm still wondering why this approach versus shipping Node API bindings to |
|
We might decide to change the names in the future. But this is essentially providing bindings to dbc without needing to use cgo/FFI (maybe rename the current In the future we might make the package available in the browser too, but we wouldn't be able to provide any driver installation/uninstallation in the browser and it might make more sense to just provide a simple sdk to the CDN rather than making this available in the browser. The primary purpose of the wasm package is to maintain the logic for handling driver installation/uninstallation without having to reproduce it |
|
Sounds good, thanks for explaining. PS: I think we want to have whatever the primary package for dbc is be exactly |
Summary
Adds a WebAssembly (
GOOS=js GOARCH=wasm) build ofdbcplus a Node-importable npm package,@columnar-tech/dbc-wasm, so applications can search registries, resolve versions, install/uninstall ADBC drivers to disk, and verify signatures in-process — without spawning the CLI subprocess.What's included
wasm/): fetch-backed HTTP transport (roundtripper_js.go), a Promise bridge (bridge_js.go), the JS API surface (api_js.go,ops_node_js.go), and Node (-tags dbcnode) + browser entrypoints.http.DefaultClient/dbc.DefaultClientare routed through the fetch transport so auth-internal OAuth refresh works under wasm.packages/npm-wasm/): CJS/ESM/TS loaders, a shared Go/wasm bootstrap (boot.cjs), and one canonical client surface (search/resolve/install/uninstall/listInstalled/verifySignature/close).baseURL/credentialare isolated betweenloadDbc()instances and refreshed tokens persist across calls.loadDbc({ worker: true })) running the runtime in aworker_threadsWorker, with a robust lifecycle: idempotentclose(), init-failure teardown (no leaked worker), and RPC rejection on close/exit instead of hanging.normalizeLocation,splitConfigListseam) — experimental..github/workflows/wasm.yml): cross-OS (ubuntu/macos/windows) build + smoke + worker-smoke + unit tests.Phasing
Phase 0 (de-risk spike — see
wasm/spike/REPORT.md) → Phase 3 (npm package) → Phase 4 (response streaming, Windows groundwork, worker backend). Review findings from each phase were addressed in focused follow-up commits.Testing
node packages/npm-wasm/test/{smoke,worker.test,normalize.test}.cjs— all pass (in-process, worker, and path-normalization).GOOS=js GOARCH=wasmwith and without-tags dbcnode).Known limitations / follow-ups
packages/npm-wasm/index.d.tsis the single canonical type declaration.windows-latestCI job exercises the drive-rootos.MkdirAll/os.OpenRootround-trip as a non-continue-on-errorsignal.install/listInstalleduse the process-global platform (no per-call override);verifySignaturetrust is anchored toSignedByColumnarby design.